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Introduction 

Servicemembers  who  suffer  combat,  training,  or  accidental  injuries  that  damage  their 
sensory  capabilities  can  have  great  difficulty  returning  to  productive  lifestyles  once  healed  from 
their  initial  trauma.  Sudden  loss  of  vision  can  overwhelm  a  previously  healthy  individual’s  ability 
to  interact  with  the  world,  adversely  impact  the  individual's  recovery  from  physical  and  emotional 
trauma,  and  prevent  a  return  to  the  community  as  a  productive,  stable  member  of  society.  This 
project  seeks  to  advance  technologies  for  non-invasive  vision  sensory  substitution  and 
augmentation  in  order  to  allow  these  individuals  to  return  to  more  normal,  healthy  social 
interactions.  This  project  exploits  two  specific  capabilities  of  the  human  central  nervous  system, 
namely  cross-modal  sensory  interactions  and  brain  plasticity,  to  address  the  needs  of  these 
injured  servicemembers.  Perception  takes  place  in  the  brain,  not  at  the  end  organ  (Bach-y-Rita, 
1972);  therefore,  the  brain  can  learn  to  reinterpret  the  meaning  of  signals  from  specific  nerves 
(e.g.,  from  tactile  receptors)  given  appropriate  self-generated  feedback.  This  forms  the  basis  for 
interfaces  that  can  non-invasively  and  unobtrusively  use  alternative  sensory  pathways  to 
provide  information.  The  Florida  Institute  for  Human  and  Machine  Cognition  (IHMC)  will  develop 
a  proof  of  concept  prototype  Anthro-Centric  Multisensory  Interface  for  Vision 
Augmentation/Substitution  (ACMI-VAS)  system.  We  envision  that  with  appropriate  development, 
the  ACMI-VAS  concept  could  be  integrated  and  reduced  in  size  to  provide  a  robust  situation 
awareness  (SA)  of  visual  information  during  activities  of  daily  living  (ADLs). 

Body 

Traumatic  brain  injury  (TBI)  can  also  induce  loss  of  one  or  more  sensory  capabilities. 
Intermittent  or  permanent  loss  of  veridical  sensory  information  adversely  affects  SA,  leading  to 
an  inability  to  perceive  and  comprehend  the  meaning  of  elements  in  the  environment  and  to 
project  their  future  states.  Without  accurate  SA,  one’s  ability  to  interact  in  a  dynamically 
changing  world  diminishes  (Endsley,  2000).  The  sequelae  of  TBI  and  somatic  polytrauma 
suffered  by  this  growing  population  may  evolve  over  many  months  after  injury,  manifesting  as 
significant  loss  of  one  or  more  sensory  channels  well  after  a  traumatic  event  (Owens,  2008). 
While  current  technologies  for  noninvasive  sensory  substitution  provide  an  inadequate 
replacement  for  lost  sensory  capabilities,  they  can  augment  residual  sensation.  In  addition, 
when  integrated  with  other  substitution  technologies,  these  technologies  can  improve  SA,  and 
therefore  open  opportunities  to  injured  servicemembers  that  they  might  not  otherwise 
investigate.  This  research  and  development  project  seeks  to  leverage  a  number  of  current 
technologies  as  well  as  some  that  are  under  development  into  a  novel  multi-sensory  vision 
augmentation/substitution  interface  that  will  enable  wounded  servicemembers  to  regain  some 
measure  of  normal  visual  interactions  and  functional  return  to  ADLs. 

Sensory  loss  can  be  addressed  by  using  precisely  positioned  large  magnetic  fields 
(Kupers  et  al.,  2006)  or  surgically  with  implanted  devices  like  cortical  (Dobelle,  2000;  Fernandez 
et  al.,  2005)  direct  nerve  implants  (Chai,  et  al.  2008)  and  end  organ  stimulators,  such  as 
cochlear  or  retinal  implants  (Zhou  &  Greenberg,  2005;  Weiland,  Liu,  &  Humayun  2005;  Veraart 
et  al.,  2003;  Maynard,  Nordhausen,  &  Normann,  1997;  Rauschecker  &  Shannon,  2002),  but  this 
adds  both  surgical  trauma  and  risk  of  infection  (Reefhuis  et  al,  2003).  A  recent  Australian 
government  initiative  provided  funding  for  the  development  of  both  supra-choroidal  and  direct 
cortical  (VI)  stimulation  arrays.  The  initiative’s  supra-choroidal  development  plan  includes  an 
initial  low  density  98  element  array  followed  by  an  eventual  high  density  1000  element  array 
(Ong  and  da  Cruz,  2011).  These  devices  require  chronic  indwelling  implants  (Figure  1)  that,  at 
the  current  state  of  the  art,  provide  limited  resolution  (e.g.,  to  20/120  equivalent  vision,  Dobelle, 
2000:  Ahuja,  et  al;  2011)  and,  in  the  case  of  cortical  implants,  may  induce  seizures  (Javaheri  et 
al.,  2006;  Kotler,  2002).  While  technological  advances  have  and  will  continue  to  dramatically 
increase  resolution  available  from  retinal  implants  (Zrenner  et  al.,  2010;  Stingl,  et  al,  2013),  they 
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still  require  invasive  surgery,  indwelling  foreign  bodies  and  an  intact  oculus.  Given  the  rapid 
development  of  modern  electronics,  wireless  communications,  batteries,  and  computer 
technologies,  it  stands  to  reason  that  implanted  devices  will  also  continue  to  improve 
qualitatively  rapidly  and  patients  who  receive  implants  will  need  to  seriously  consider  the  risks 
and  benefits  of  “upgrading”  their  prostheses  through  more  surgery.  Servicemembers  blinded 
from  battlefield  polytrauma  will  have  likely  endured  multiple  surgical  procedures  by  the  time  they 
have  stabilized  sufficiently  to  consider  a  retinal  or  cortical  implant  and  may  not  be  candidates 
due  to  injury  to  underlying  neural  tissue  (retina),  pathways  (optic  nerve),  or  cortical  processing 
(occipital  lobe).  An  alternative  solution  to  this  form  of  sensory  replacement,  namely  sensory 
substitution,  has  shown  promise  for  blind  individuals  and  can  tolerate  damage  in  any  of  these 
components  that  make  up  the  “retinex”  (Land,  1964)  normally  required  for  human  perception  of 
the  visual  environment. 


Figure  1:  (left)  Photograph  of  the  Argus™  I  and  II  Retinal  Prosthesis  System  epiretinal  microelectrode 
arrays  (Second  Sight  Medical  Products,  Inc,  Sylmar,  CA)  recently  approved  for  use  in  the  United  States 
(note  increase  in  electrode  density  from  16  to  60  electrodes  between  Argus  I  and  Argus  II).  (center) 
Retinal  Implant  AG  (Reutlingen,  Germany)  subretinal  combined  photodiode/electrode  array  with  1500 
sensors  and  electrodes,  (right),  X-ray  of  implanted  cortical  array  (Dobelle,  2000)  positioned  on  the 
occipital  cortex. 

Perception  takes  place  in  the  brain,  not  at  the  end  organ  (Bach-y-Rita,  1972);  therefore, 
the  brain  can  learn  to  reinterpret  the  meaning  of  signals  from  specific  nerves  (e.g.,  from  tactile 
receptors)  given  appropriate  self-generated  feedback.  This  forms  the  basis  for  interfaces  that 
can  non-invasively  and  unobtrusively  use  alternative  sensory  pathways  to  provide  information. 
This  means  that  the  information  displayed  does  not  necessarily  need  to  represent  the 
underlying  data  at  high  resolution,  rather  abstract  representations  of  the  sensory  environment 
information  can  provide  sufficient  data  for  operator  decision  making  and  improved  SA  (Raj, 
Kass,  &  Perry,  2000).  Such  sensory  substitution  mechanisms  exploit  the  plasticity  inherent  to 
the  brain  and  nervous  system,  supporting  both  long  term  and  short  term  anatomical  and 
functional  remapping  of  sensory  data  (Finkel,  1990;  Walcott  &  Langdon,  2001).  Sensory 
substitution  refers  to  the  remapping  of  sensory  data  from  the  normal  sensory  receptor  field  for  a 
particular  type  of  data  to  other  channels  of  information  perception  (Bach-y-Rita,  Collins, 
Saunders,  White,  &  Scadden,  1969).  With  appropriate  feedback,  the  plasticity  of  the  human 
brain  allows  individuals  to  learn  to  perceive  the  substituted  data  with  little  cognitive  effort, 
especially  if  training  is  provided  soon  after  sensory  loss  (Bavelier  &  Neville,  2002).  For  maximal 
benefit,  the  interface  must  also  intelligently  pre-process  the  incoming  data  to  account  for 
differences  in  capability  between  the  alternative  channels  and  the  ones  normally  used  to 
perceive  given  sensory  data,  as  well  as  provide  intuitive  control  and  data  management.  Many 
sensory  substitution  devices  and  approaches  have  been  developed  over  the  past  decades 
(Machts,  1920).  Modern  computer  and  electronic  design,  however,  now  enables  the 
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development  of  intelligent,  noninvasive  interfaces  unobtrusive  enough  for  use  in  everyday 
activities. 

Neural  projections  from  the  visual,  auditory  and  proprioceptive  sensory  systems  interact 
in  the  brainstem  at  the  superior  colliculus  (Meredith  &  Stein,  1986;  Wallace  &  Stein,  2000)  and 
higher  levels  (Fort  et  al.,  2002),  where  mechanisms  such  as  non-synaptic  diffusion 
neurotransmission  (as  well  as  synaptic  connections)  can  cross-modally  engage  different 
sensory  channels  (Bach-y-Rita,  1995).  In  addition,  crossmodal  activity  has  been  demonstrated 
in  the  primary  and  secondary  cortical  processing  areas  for  individuals  with  and  without  sensory 
impairment  (Schroeder  &  Foxe,  2005).  For  example,  deaf  individuals  show  activation  in  the 
auditory  cortex  in  response  to  visual  stimuli  (Finney,  Fine,  &  Dobkins,  2001),  blind  individuals 
show  visual  cortex  activity  in  response  to  touch  (Sadato  et  al.,  1996;  Sadato  et  al.,  2004; 
Sathian,  2005)  and  vibratory  stimuli  (Burton,  Sinclair,  &  McLaren,  2004),  and  auditory  stimuli 
can  elicit  changes  in  somatosensory  cortex  activity  (Foxe  et  al.,  2000).  The  fact  that  these 
interactions  exist  and  become  more  pronounced  given  sensory  channel  deprivation  (Sathian, 
2005)  supports  the  use  of  sensory  substitution  as  a  potential  solution  for  partial  restoration  of 
lost  sensory  channels. 

With  veridical  multisensory  information,  these  mechanisms  appear  to  enhance 
perception  accuracy  and  reaction  time  (Deiderich,  1995)  as  well  as  modulate  ongoing  cognitive 
processes  (Schroeder  &  Foxe,  2005)  while  improving  workload  performance  and  SA  (Wickens 
&  Holland,  1999).  By  exploiting  this  cortical  crossmodal  integration  (Calvert,  2001),  we  and  other 
researchers  have  shown  that  individuals  with  sensory  loss  due  to  artificial  restrictions,  disease, 
congenital  defect,  or  injury  can  use  sensory  substitution  interfaces  to  exploit  this  inherent 
plasticity  of  the  brain  and  nervous  system  for  both  long  term  and  short  term  anatomical  and 
functional  remapping  of  sensory  data  (Walcott  &  Langdon,  2001;  Ptito,  Moesgaard,  Gjedde,  & 
Kupers,  2005;  Kaczmarek,  Bach-y-Rita,  Tompkins,  &  Webster,  1985)  and  improvement  in  SA 
(Raj,  Kass,  &  Perry,  2000;  Saunders,  Hill,  &  Franklin,  1981).  Recent  brain  imaging  studies  have 
confirmed  crossmodal  modulation  of  activity  across  different  sensory  cortices  that  vary 
depending  on  whether  multiple  sensory  channels  provide  congruent  information  or  incongruent 
information  (Johnson  &  Zatorre,  2005;  Jones  &  Callan,  2003;  Fort  et  al.,  2002;  Laurienti  et  al., 
2002).  Because  attentional  resources  or  cognitive  effort  required  to  process  the  sensory 
channel  data  also  manifests  as  intersensory  cortical  activity  modulation  (Loose  et  al.,  2003; 
Kanwisher  &  Wojciulik,  2000;  van  Wassenhove,  Grant,  &  Poeppel,  2005;  Mozolic  et  al,  2008), 
multisensory  aids  for  the  blind  must  minimize  sensory  display  complexity  to  reduce  cognitive 
demands. 

A  number  of  options  for  visual  sensory  substitution  exist,  however,  the  two  most 
investigated  methods  use  audio  and  tactile  representations,  respectively,  of  visual  information. 
The  Braille  tactile  alphabet  represents  the  most  common  example  of  tactile  substitution  using 
two  columns  of  six  or  eight  raised  bumps  either  embossed  on  paper  or  presented  using  a 
mechanical  device.  While  this  has  worked  well  for  plain  text  transcriptions,  it  becomes  unwieldy 
for  graphical  information  for  technical  information  such  as  mathematical  formulas  (Mogo  & 
Archambault,  2003)  and  the  graphically  rich  visuals  of  magazines,  the  Internet  and  the  natural 
world.  (Boehm,  1986;  Ifukube,  Sasaki  &  Peng,  1991).  An  alternative  technique  promoted  by 
Meijer  (1992)  called  “The  vOICe“  system  converts  images  into  an  audio  time  multiplexed 
frequency  and  amplitude  representation  that  sweeps  across  successive  frames  of  video  to 
encode  spatial  information  into  a  complex  audio  stream  (Amedi  et  al.,  2005).  The  biggest 
drawback  to  these  systems  related  to  the  amount  of  training  required  to  become  fluent  in  the 
alternative  representation  and  the  fact  that  these  methods  heavily  engage  the  remaining 
senses.  With  Braille,  the  reader’s  hands  are  unable  to  perform  other  ADLs  (such  as  making  a 
sandwich).  The  vOICe  system  audio  display  and  computerized  text  to  speech  screen  readers 
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such  as  Job  Access  With  Speech  (JAWS®,  Freedom  Scientific,  Inc.,  St.  Petersberg,  FL)  (Bryant 
&  Bryant,  2003)  as  well  as  more  recent  multimodal  computer  interfaces  for  the  blind  (Yu  et  al., 
2006)  require  significant  auditory  cognitive  engagement  and  may  prevent  accurate  sensing  of 
events  in  the  ambient  acoustical  environment  (e.g.,  a  ringing  telephone).  Portable  camera 
based  readers  such  as  the  knfbReader  Mobile  (K-NFB  Reading  Technology,  Inc.,  Newton 
Lower  Falls,  MA)  allow  the  blind  to  carry  optical  character  recognition  into  the  real  world,  but 
camera  resolution  and  lack  of  context  of  the  visual  environment  (Gaudissart  et  al.,  2005)  make 
such  systems  cumbersome  to  use  in  practice  (e.g.,  the  user  does  not  have  a  sense  of  the  size 
of  the  type  or  if  there  is  any  text  to  recognize  without  additional  input  from  a  sighted  individual). 

The  Florida  Institute  for  Human  and  Machine  Cognition  (IHMC)  has,  therefore,  focused 
on  integration  and  development  of  non-invasive  methods  of  presenting  visual  information  to 
blinded  servicemembers  using  sensory  substitution.  By  leveraging  previous  work  for  the  United 
States  Navy  and  the  National  Aeronautics  and  Space  Administration  (NASA)  originally  designed 
to  enhance  SA  for  aviators  and  astronauts,  along  with  modern  tactile  technologies  designed  for 
the  blind,  we  have  developed  prototype  hardware  components  that  could  provide  a  substantial 
level  of  visual  sensory  information.  With  early  versions  of  these  components,  we  have  provided 
partial  demonstrations  to  and  solicited  early  feedback  from  four  recently  blinded  military 
servicemembers.  These  evaluators  suffered  polytraumatic  injuries  that  resulted  in  enucleation  of 
both  eyes  10-48  months  prior  to  participation.  All  had  and  at  least  one  ocular  prosthetic  fitted. 
Two  existing  tactile  interfaces  were  demonstrated,  the  U.  S.  Navy/NASA/IHMC  Tactile  Situation 
Awareness  System  (TSAS)  and  the  video  camera  based  Wicab,  Inc.,  (Middleton,  Wl)  BrainPort® 
Wearable  Aid  for  Vision  Enhancement  (BP-Wave  II).  The  former  provided  a  limited  sense  of 
peripheral  visual  object  detection  using  24  electromechanical  vibro-tactile  transducers  (tactors) 
mounted  in  a  garment  that  creates  a  three-dimensional  tactile  torso  interface  (TTI)  array  on  the 
body.  The  latter  consisted  of  an  18x18  array  of  electrotactile  tactile  transducers  placed  on  the 
user’s  tongue  in  a  two  dimensional  array,  now  available  as  a  20x20  array  for  clinical  research  or 
as  a  25x25  prototype  (Figure  2). 


Figure  2:  Blind  servicemembers  using  prototype  sensory  substitution  interfaces,  (left)  Using  the  TTI  to 
receive  peripheral  attention  directing  tactile  cues,  (center  left)  Using  the  BP-WAVE  II  to  negotiate  stairs 
without  assistance  and  (center  right)  to  read  text,  (right)  BrainPort®  Intraoral  Device  (IOD)  electrotactile 
tongue  array  (-600  active  tactile  pixels). 

With  these  systems  we  have  demonstrated  awareness  of  nearby  objects  on  the  torso 
and,  through  the  BrainPort®  tongue  array,  identification  of  shapes,  shape  orientations,  reading 
(up  to  4-5  word  sentences),  catching  balls  rolled  across  a  table,  navigating  unfamiliar  office 
spaces,  negotiating  stairs,  identifying  open  parking  stalls  from  ones  with  vehicles,  performing 
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standard  visual  acuity  tests  to  20/40  equivalent  using  standard  eye  charts  and  recognizing 
family  members  (Figure  3).  Each  individual  trained  to  perform  these  tasks  in  less  than  4-6 
hours. 


Figure  3:  Initial  BP-WAVE  II  activities  performed  during  technology  demonstrations  with  recently  blinded 
servicemembers.  left  to  right,  top  to  bottom,  Catching  balls,  navigating  office  spaces,  locating  open 
parking  stalls,  noticing  infant’s  hair,  and  reading  eye  charts  to  20/80  (images  used  with  permission). 


In  addition,  IHMC  has  demonstrated  that  a  direct  connection  between  the  tactile 
interface  and  a  computer  graphical  display  can  provide  a  superior  qualitative  experience  by 
bypassing  the  limitations  of  cameras  (e.g.,  glare,  changing  lighting  conditions,  focus  and 
unintentional  movements  of  the  head).  While  this  direct  connection  removes  the  need  to 
position  the  user  in  front  of  a  computer  (a  useful  capability  for  those  with  orthopedic  or  other 
injuries  that  prevent  prolonged  upright  posture),  we  have  prototyped  two  methods  of  tactile 
computer  interactions  that  mimic  sighted  interactions.  Using  video  oculography,  we  have 
demonstrated  that  blind  individuals  can  use  extraocular  musculature  to  control  insensate  eyes 
(even  prosthetic  ones)  with  sufficient  accuracy  to  pan  and  scan  to  read  across  an  image 
presented  tactually.  Likewise,  head  position  can  control  image  zoom  such  that  a  blind  user  can 
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lean  toward  or  away  from  an  object  of  interest  to  zoom  in  or  out.  In  the  current  implementation, 
looking  away  from  the  computer  screen  automatically  switches  to  direct  camera  feed,  which 
allows  the  user  to  look  down  at  the  keyboard  when  typing  or  observe  other  items  in  the 
environment.  Alternatively,  we  have  also  implemented  a  touch  screen  mechanism  that  allows 
the  user  to  feel  the  pixels  under  his  or  her  fingertip  via  the  tongue  while  dragging  across  the 
screen  (Figure  4).  Both  mechanisms  have  been  mastered  by  participants  in  less  than  30 
minutes,  indicating  that  the  method  does  not  significantly  task  cognitive  resources. 


eye/head  tracking  apparatus  to  determine  point  of  gaze  despite  the  users’  prosthetic  eyes.  The  software 
determines  if  the  participant  is  looking  at  the  screen  and  returns  a  100-200  pixel  sample  of  the  screen 
image.  The  user  intuitively  controls  zoom  level  by  leaning  either  toward  or  away  from  the  screen.  Looking 
down  automatically  selects  a  gaze  directed  subsample  of  the  video  image  from  the  head  mounted 
camera  (allowing  the  user  to  find  keys  on  the  keyboard),  right,  When  the  user  touches  the  screen,  the 
subsection  of  the  screen  image  underneath  the  fingertip  is  presented  to  the  tongue  instead. 

Two  major  issues  were  noted  with  the  prototype  system  evaluations,  namely  registration 
of  gaze  position  relative  to  the  visual  task  and  perception  of  changes  in  the  environment  outside 
the  field  of  view  of  the  camera.  When  reading,  for  example,  inadvertent  head  and  body 
movements  caused  the  camera  image  to  move  away  from  the  word  or  letter  of  interest. 
Reacquisition  of  the  word  or  letter  required  additional  panning  and  scanning  in  order  to  find  the 
line  of  text  and  the  word  or  letter.  The  lack  of  peripheral  visual  sensation  appeared  to  cause  a 
high  level  of  anxiety  for  recently  blinded  individuals  (who  may  not  have  yet  adapted  to  their 
visual  loss).  Blind  individuals  are  often  startled  when  others  approach  quietly  and  begin  talking 
to  or  make  physical  contact  with  them.  These  effects  would  still  occur  even  if  the  IOD  received 
data  from  prosthetic  eyes  with  embedded  cameras  (figure  5),  due  to  the  lack  of  visual  context 
and  peripheral  vision. 
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Figure  5:  Electronics  (left)  embedded  in  the  camera-enabled  prosthetic  eye  (right)  developed  by  the 
Eyeborg  Project  (Toronto,  Canada).  Downloaded  from  http://eyeborgproiect.com. 

These  issues  warranted  the  integration  of  additional  displays  and  sensors  to  provide 
paracentral  and  peripheral  visual  information  and  a  sense  of  gaze  location  within  the  broader 
visual  context.  Following  the  initial  technology  demonstrations,  IHMC  aggregated  these 
prototypes  in  order  to  improve  a  user's  tactile  visual  sensory  substitution  perception.  Using  the 
BP-WAVE-II  prototype  and  the  previous  generation  VideoTact,  we  determined  that  tasks  that 
required  serial  visual  scan,  such  as  reading,  visual  acuity  testing,  identifying  shapes  arrange  in 
rows  and  columns,  etc.,  could  be  performed  more  easily  when  using  the  two  tactile  interfaces 
simultaneously  (Figure  5). 


Figure  5:  Initial  mock  up  of  ACMI-VAS  for  foveal  and  paracentral  vision  substitution.  Image  (upper  right 
inset)  presented  on  the  VideoTact  (upper  left  inset)  is  zoomed  in  less  than  the  BrainPort®  image  zoom 
level  (center  display  of  lower  right  inset)  to  provide  a  slightly  wider  field  of  view.  The  physical  size  of  the 
components  has  been  reduced  significantly  and  control  of  the  combined  technologies  has  also  been 
simplified  using  IHMC's  AMI  architecture. 

IHMC  used  its  existing,  in-house  developed  Adaptive  Multiagent  Integration  (AMI) 
software  architecture,  which  can  easily  connect  components  such  as  displays,  sensors, 
algorithms  and  adaptive  automation  via  a  standardized  Java  (Sun  Microsystems,  Inc,  Santa 
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Clara,  CA)  interface.  AMI  associates  explicit  ontological  definitions  with  each  agent  that  allow 
rapid  integration  of  new  components  (as  software  agents)  into  the  architecture  because  the 
system  identifies  and  makes  data  connections  between  agents  automatically,  based  on  data 
types  and  relative  quality  of  similar  data  streams.  The  large  complement  of  devices,  including 
video,  motion  capture,  tactile  and  audio  interfaces,  pressure,  orientation  and  psychophysiologic 
sensors,  previously  integrated  as  software  agents  (Johnson  et  al.,  2005)  were  leveraged  and 
additional  agents  were  created  as  needed.  The  architecture  is  inherently  scalable,  using 
available  processing  power  and  allows  unlimited  nodes  and  agents  on  wired  or  wireless 
networks.  Communications  between  agents  are  made  peer  to  peer  and  support  high  data  rates 
and  both  secure  and  open  transport  mechanisms.  Sensor  signals  or  processed  data  can  be 
delivered  to  multiple  displays  across  different  modalities  (e.g.,  video,  audio,  vibrotactile, 
electrotactile,  etc.)  simultaneously  or  separately,  enabling  side-by-side  real-time  evaluation  of 
various  sensory  substitution  implementations  (Raj  et  al.,  2005).  Leveraging  AMI  for  this  project 
enabled  rapid  development  of  the  ACMI-VAS  prototype  by  taking  advantage  of  existing  software 
agents  for  a  number  of  tactile  interfaces  (including  torso,  abdominal  and  tongue  placed 
displays),  various  video  cameras,  peripheral  range  and  bearing  sensing,  as  well  as  head  and 
eye  tracking  (e.g.,  OptiTrack,  NaturalPoint,  Inc.,  Corvallis,  OR;  RK-826y-BPCI,  IScan,  Inc., 
Burlington,  MA). 

Earlier  in  this  project  we  developed  a  color  filter  assembly  that  mounts  to  an  unmodified 
BrainPort®  V-100  vision  device  that  has  a  400  pixel  intraoral  device  (IOD)  or  to  an  earlier 
BrainPort®  BP-WAVE-II  with  a  600  pixel  IOD.  We  integrated  a  Playstation3  controller  into  a 
button  pad  interface  box  to  reduce  the  number  of  items  the  user  will  have  to  manipulate. Three 
blind  individuals  evaluated  this  system  and  could  identify  primary,  secondary  and  tertiary  colors 
within  15-20  minutes  (Figure  6). 


Figure  6:  Research  staff  (using  BrainPort®)  changing  filter  (black  disk  mounted  to  glasses)  with 
Playstation3  game  controller  buttons  to  identify  color  of  illuminated  button  pad. 


By  leveraging  other  funding  vehicles  following  the  initial  technology  demonstrations 
noted  above,  the  ACMI-VAS  system  benefits  from  advances  in  the  BrainPort®,  the  VideoTact 
and  TTI  (Figure  7).  Notably,  IHMC  has  acquired  two  BrainPort®  VI 00  (20x20  pixel)  units  to 
replace  the  prototype  BP-WAVE-II  units.  We  have  collaborated  with  ForeThought  Development, 
LLC,  to  design  a  more  compact  VideoTact  system  that  utilizes  modern  electronics  components. 
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Figure  7:  (left)  Current  BrainPort  VI 00  system  showing  sunglasses  mounted  camera,  handheld 
embedded  processor/user  input  device  and  400  tactor  IOD.  (center)  TTI  with  ACU  mounted  C-2  tactors 
and  (right)  comparison  of  C-2  (red)  and  C-3  (black)  tactor  sizes. 


Lastly,  we  have  developed  a  reflected  infrared  sensor  range  and  direction  detection 
system  that  can  detect  objects  in  the  environment  and  represent  their  locations  and  motion 
using  the  TTI  (Figure  8),  which  can  use  the  much  smaller,  lighter  weight  C-3  tactors  versus  the 
larger,  older  C-2  tactors  (Engineering  Acoustics,  Inc.,  Casselberry,  FL). 


Figure  8:  (left)  IHMC's  revised  embedded  microprocessor  controlled  24  channel  reflected  infrared 
emission  range  and  direction  peripheral  vision  sensor  PC-104  size  circuit  board  with  tactor  drive 
capability,  (center  left)  Schematic  of  head  worn  cap  with  embedded  IR  emitters  and  receivers  detecting 
(center  right)  objects  in  the  environment,  (left)  A  second  circuit  board  receives  the  signal  and  represents 
the  information  on  the  torso  using  the  TTI. 

These  technology  advances  formed  the  basis  of  the  ACMI-VAS  central/paracentral  and 
peripheral  vision  substitution  concept.  Because  the  VI 00  is  now  available  as  an  investigational 
device  for  research  applications,  IHMC  designed  a  study  to  compare  blind  performance  on 
ADLs  with  the  BrainPort®  alone,  or  with  ACMI-VAS  (which  includes  the  BrainPort®  for  foveal 
substitution,  the  VideoTact  for  parafoveal  substitution,  and  the  TTI  for  peripheral  substitution). 

In  the  ninth  project  quarter,  the  project  received  a  no  cost  extension  through  3QTR 
FY13.  The  research  staff  continued  practicing  the  test  protocol,  which  had  been  approved  by 
the  U.  S.  Army  Human  research  Protections  Office  (HRPO)  in  the  final  month  of  the  prior 
funding  year.  Under  another  funding  vehicle,  we  also  completed  the  design  and  fabrication  of 
the  initial  miniaturized,  battery  powered  TTI  driver. 

During  this  tenth  reporting  period,  effort  continued  on  hardware  integration  for  the 
multisensory  “haptic  retina”  system  based  on  feedback  from  research  staff  practicing  the 
protocol  utilizing  the  new,  smaller  TTI  driver  system  with  the  lighter  weight  C-3  tactors.  The 
garment  with  these  lighter  tactors  has  been  deemed  more  comfortable  that  the  older  C-2  Based 
vest  by  the  research  staff.  Similarly,  Forethought  Development,  LLC,  continued  fabrication  of 


Approved  for  public  release;  distribution  unlimited  12 


the  ETv6  miniaturized  VideoTact,  however,  component  availability  became  a  greater  issue  as 
the  higher  voltage  parts  necessary  for  electrodermal  stimulation  were  discontinued  as 
commercial  electronics  demand  changed  to  lower  voltage  components.  This  modern  unit  will  be 
much  lighter  and  easier  for  participants  to  use.  During  this  period  we  also  increased  the  color 
gamut  available  on  the  color  identification  task.  This  results  in  a  visible  flicker  of  the  LEDs  that  is 
not  perceptible  on  the  tongue.  This  allows  us  to  more  accurately  generate  specific  colors. 

In  the  eleventh  reporting  period,  Forethought  Development,  LLC,  completed  the 
fabrication  of  the  ETv6  miniaturized  VideoTact  array  and  driver  circuit  design  and  has  begun 
work  on  the  hardware  controller  for  the  miniaturized  array.  During  this  reporting  period,  the  PI 
travelled  to  the  Military  Vision  Symposium  in  Boston,  MA,  in  September  2012.  At  this  forum,  the 
PI  presented  the  current  state  of  the  ACMI-VAS  project  and  the  approach  to  be  used  in  the 
human  research  participant  testing. 

Relationship  to  award  Statement  of  Work 

At  the  end  of  the  third  year  of  the  program,  we  have  developed  a  multisensory  interface 
that  provides  visual  substitution  via  tactile  displays  that  provides  a  partial  functional  restoration 
of  visual  perception.  Based  on  the  original  statement  of  work,  we  have  completed  the  following 
tasks: 

Specific  Aim  #  1:  Develop  a  multi-sensory  tactile  interface  to  augment  or  substitute  for  recently 
acquired  visual  impairment 

Task  1.  Integrate  IHMC  Tactile  interfaces  for  haptic  retina  application 

la.  Finalize  sensor  integration  plan  for  peripheral  and  central  vision  functions 

lb.  Develop  integrated  abdomen,  torso  and  tongue  display  mounting  system 

lc.  Develop  multiple  sensor  hardware  mounting  system 

ld.  Develop  software  functional  requirements 
Milestone  #1:  ACMI-VAS  optical  design  review 

Task  2.  Integrate  visual  sensory  augmentation/substitution 

2a.  Evaluate  feasibility  of  providing  stereoscopic  vision  with  current  interfaces 
2b.  Define  paracentral  display  icon  for  registration  of  foveation  point 
2c.  Fabricate  REVISED  hardware  for  IHMC  peripheral  vision  sensor 

Specific  Aim  #2:  Evaluate  ACMI-VAS 

Task  3.  Code  software  for  visual  environment  interactions 

3a.  Integrate  central  (foveal),  paracentral  and  peripheral  vision  manual  controls 

3b.  Develop  calibration  procedures  for  ACMI-VAS 

3c.  Code  user  interface 

3d.  Code  performance  evaluation  interface 

3e.  Test  and  evaluate  interface  function  with  tactile  and  audio  displays 
3f.  Define  performance  metrics  and  code  software  for  evaluation  of  metrics 
3g.  Verify  evaluation  software  functionality 

Milestone  #3:  ACMI-VAS  haptic  retina  evaluation  system  design  complete 

Task  4.  Demonstration  and  evaluation 

4a.  Submit  IRB  and  HRPO  applications  for  use  of  human  research  participants 
4b.  Prototype  system  functional  verification  and  testing 

Remaining  Tasks/Milestones: 

Task  2.  Integrate  visual  sensory  augmentation/substitution 

2d.  Complete  Integration  of  software  (using  AMI)  of  IHMC  peripheral  vision  sensor 
2e.  Test  &  evaluate  final  ACMI-VAS  haptic  retina  software/hardware  integration 
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Milestone  #2:  Complete  ACMI-VAS  haptic  retina  prototype  system 


Task  4.  Demonstration  and  evaluation 

4c.  Identify  and  recruit  20  recently  blinded  research  participants 

4d.  Human  participant  testing 

4e.  Human  participant  data  analysis 

Milestone  #4:  Collect  and  analyze  ACMI-VAS  human  research  participant  data 
Milestone  #5:  Draft  manuscript  for  submission  to  Ophthalmology  (Journal} 

Deliverables:  1)  HRPO  application  renewal,  2)  Quarterly  reports,  and  3)  Annual  Report 
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Key  research  accomplishments  (project  years  1  through  3) 

•  We  successfully  integrated  a  prototype  haptic  retina  via  the  AMI  software  architecture. 

•  We  determined  that  serial  visual  tasks  such  as  reading  and  identifying  shapes  on  a  grid  was 
enhanced  when  using  ACMI-VAS. 

•  We  determined  that  colors  could  be  identified  tactually. 

•  We  developed  a  test  protocol  to  evaluate  the  effectiveness  of  the  ACMI-VAS  concept 
against  the  BrainPort®  VI 00  alone. 

•  We  received  HRPO  approval  to  begin  human  research  participant  testing. 

•  We  fabricated  a  miniaturized  TTI  driver  system 

Reportable  outcomes 

Received  HRPO  approval  for  use  of  human  research  participants. 

Conclusion 

During  years  one  through  three  of  the  ACMI-VAS  project,  we  developed  a  prototype 
ACMI-VAS  system  and  the  research  environment  needed  to  evaluate  it.  We  improved  the  user 
control  interfaces  and  developed  a  method  to  allow  tactual  understanding  of  color.  The  final 
portion  of  this  grant  will  focus  on  human  research  participant  testing  and  evaluation,  data 
analysis,  drafting  a  publication  detailing  the  results,  and  development  of  the  final  ACMI-VAS 
prototype  design  specification  document.  The  noninvasive  nature  of  the  ACMI  approach 
ensures  that  injured  servicemembers  could  benefit  from  future  upgrades  as  technologies 
improve  (in  out-years)  without  risks  of  further  surgeries  or  infection  associated  with  implantable 
devices.  The  proposed  complementary  interface  displays  can  be  tailored  to  suit  the  needs  of  an 
individual.  For  example,  an  injury  that  spared  the  peripheral  vision  may  only  require  the  higher 
resolution  displays,  whereas  a  condition  like  hemianopsia  might  only  require  a  low-resolution 
spatial  awareness  component.  This  proposed  technology  development  will  result  in  a  single 
integrated  system  prototype  capable  of  providing  an  alternative  mechanism  for  visual  sensing  of 
high  resolution  foveal  vision,  low  resolution  peripheral  vision  and  stabilization  of  the  imagery 
despite  perturbations  of  the  head.  Even  profoundly  blind  individuals  would  benefit  from  the 
modularity  of  the  system  as  they  could  choose  to  use  specific  displays  for  any  given  activity. 
The  use  of  the  AMI  software  agent  framework  ensures  that  integration  of  improvements  in  any 
of  the  major  technologies,  including  sensing  devices  (e.g.,  cameras)  and  interfaces  (potentially 
even  implantable  ones)  will  occur  quickly,  speeding  up  evaluation  of  incremental  changes  and 
their  deployment  to  the  users. 
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Appendix:  Acronyms 


•  ACMI-VAS  -  Anthro-Centric  Multisensory  Interface  for  Vision  Augmentation/Substitution 

•  ADL-  Activities  of  Daily  Living 

•  AMI  -  Adaptive  Multiagent  Integration 

•  BP-WAVE  II-  BrainPort®  Wearable  Aid  for  Vision  Enhancement,  version  II 

•  FrACT  -  Freiburg  Visual  Acuity  and  Contrast  Test 

•  HRPO  -  Human  Research  Protection  Office 

•  IHMC  -  Florida  Institute  for  Human  and  Machine  Cognition 

•  IOD  -  Intra  Oral  Device 

•  IRB  -  Institutional  Review  Board 

•  JAWS  -  Job  Access  With  Speech 

•  NASA  -  National  Aeronautics  and  Space  Administration 

•  RGB  -  Red,  Green,  Blue 

•  SA  -  Situation  Awareness 

•  TBI  -  Traumatic  Brain  Injury 

•  TSAS  -  Tactile  Situation  Awareness  System 

•  TTI  -  Tactile  Torso  Interface 
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